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ABSTRACT 

The behavior of approximately 50 employees learning 
to use a Los Alamos National Laboratory personnel information system 
with fixed-format and context-sensitive online help was monitored 
between October of 1984 and June of 1985. Data was gathered through 
direct observation during training workshops and during actual use, 
logging of consulting phone calls, and interviews. Information was 
sought to determine the following: (1) the type of training that 
would produce the greatest amount of learning for these users and 
their preferred learning styles; (2) how frequently they accessed 
online help; (3) the number of consulting calls that could have been 
answered by online help and the learning styles of the frequent 
callers; (4) ways to improve online help; and (5) whether the time 
invested in generating extensive online help was cost effective. 
Three preferred styles of learning were exhibited by users , and it 
was found that those with a guided discovery learning style were the 
most willing to access online help, users with a discovery learning 
style tended to enter an option to see what happened instead of 
accessing help, and the structured style learners were more nervous 
about using function keys and accessed online help the least. The 
context-sensitive help tended to be accessed when users were in 
trouble, and it seemed to be more frequently accessed and more useful 
to learners than the more general fixed-format help. It was also 
found that help that is context-sensitive (but not 11 intelligent" ) 
does not require major increases in software development resources. 
(DJR) 



*************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
***************************************** ****************************** 



9 

ERLC 



LA-UR -Z&'jWO*) 



BEST COPY AVAILABLE 



Los Alamos National Laboratory is operated by the University of California for the United States Department of Energy under contract W.74Q5-ENG-36. 



o 



TITLE: 



U.S. DEPARTMENT OF EDUCATION 

Otttce o( Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

4 This document has been reproduced as 
received Irom the person or organization 
originating it 

□ Minor changes have been made to improve 
reproduction quality. 

• Points of view or opinions stated in this docu- 
ment do not necessarily represent official 
OERt position or policy 



LEARNING STYLES AND EMBEDDED TRAINING: A CASE STUDY 



LU 



AUTHOR(S): 



Mary L. Stoddard 



SUBMITTED TO: 



ERIC Clearinghouse on Information Resources 



\3- 



By acceptance ol this article, the publisher recognizes that the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce 
the published form of tnts contribution, or to allow others to do so. for U.S. Government Purposes. 

The Los Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U.S. Department of Energy. 




Los Alamos National Laboratory 
Los Alamos, New Mexico 87545 



ERLC 



FORM NO 836 fl4 
ST NO 2tTi 5/81 



LEARNING STYLES AND EMBEDDED TRAINING: 
A CASE STUDY 



Mary L.Stoddard 
Los Alamos National Laboratory 

INFORM Office, MS B260 
Los Alamos, New Mexico 87545 
(505)667-9444 



ABSTRACT 

The use of online help systems as a form of training is 
recent, with the emergence of embedded training. 
Embedded training reduces the need for mode changes. 
More study of user behavior with embedded training 
systems is needed for design considerations. This 
study reports on user behavior with fixed format and 
context-sensitive online help on a Los Alamos National 
Laboratory information system and suggests that user 
learning style is a key determinant for the design of 
embedded training. 



INTRODUCTION 

The realization that online help is a form of train- 
ing for users of interactive computer systems has 
recently emerged (e.g.. Fawcette. 1983; Stoddard, 1984: 
Heise. 1985; Kearsley, 1985). There are several reasons 
for this: 

• The emergence of easier- to-use menu-driven and 
iconic interfaces has reduced the dependence on 
large printed reference manuals, 

• The discovery, through user behavior studies, that 
many people do not learn effectively with printed 
documentation. 

• The high cost of training, both in terms of the 
user's time investment and institutional costs of 
training, and management's desire to reduce train- 
ing costs. 

• The increase in the number of remote users of 
software. 

• The advancements in software design resulting in 
the ability to integrate computer-based training 
and help with the actual system. 

• The observation that mode changes required by 
separate training or help packages reduce user pro- 
ductivity. 



Kearsley (1985) defined embedded training as 
"on-line instruction which is an integral part of a sys- 
tem or product; instead of isolating learning from 
what is to be learned, embedded training makes it an 
ongoing aspect of that system or product." As examples 
of embedded training he cites helps, simulations, and 
intelligent tutors, and he fully develops a rationale for 
why embedded training is the direction to head for in 
computer-based training. Because of the recency of 
embedded training (or online help seen as a form of 
training), few studies have been performed on user 
behavior with existing embedded training systems. 
Those that have been performed do not consider user 
learning style as a critical variable, but rather 
emphasize degree of prior computer experience only 
(e.g., Cohill and Williges. 1982; studies reported by 
Houghton, 1984). 

The purpose of this paper is to describe a study of 
user behavior with fixed-format and context-sensitive 
help, which has been implemented on a personnel 
information system at Los Alamos National Labora- 
tory, and to suggest that user learning style is a key 
determinant for the design of embedded training. User 
behavior was studied for a nine-month period (October 
1984-June 1985) through direct observation, consul- 
tants' logs, and interview methodologies. 



ONLINE HELP AS EMBEDDED TRAINING 

At Los Alamos National Laboratory, a Depart- 
ment of Energy contractor with 7600 employees, a 
multi-functional and menu-driven set of information 
systems called INFORM has been developed for use by 
managers and administrators. For the more complex 
data entry systems, extensive documentation (in the 
form of walk-through quick guides) and live training 
in a computer classroom have been offered. Several 
different formats for training have been used and 
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evaluated, Including computer-based training, tradi- 
tional classroom instruction, and hands-on workshops 
(see Stoddard. 1985). 

One personnel information system, called the 
Salary Review System, has been in production since 
December 1984. This system enables Laboratory 
managers to calculate and propose raises for their 
employees, to generate reports on employee data for 
use in their decision making, and to approve raises pro- 
posed by subordinate managers, The system was 
developed in-house, is written in COBOL, uses the 
forms management system (FMS) of Digital Equipment 
Corporation, and runs on VAX/VMS machines. Exten- 
sive online help is provided throughout the system. 
This online help is accessed in two ways: 

1. Fixed-formal help: HE menu option 

The main menu of the Salary Review System con- 
tains 11 options that perform the system func- 
tions. One option is HE, which stands for H£LP. 
When HE is typed at the cursor, the users are 
prompted for the option they want a detailed 
description of. They then type in a three-letter 
option code (e.g., RRP for raise review reports, 
CWA for change write access) and are given an 
informational screen describing the function and 
usage of that option (see Figures 1 and 2). 

2. Context-sensitive help: Help key from data 
entry screens 

If users press the PF2 key (on a VT100) or the F2 
key (on an IBM PC), they receive one line of help 
information. This line either provides a descrip- 
tion of the task at hand or what the next logical 
choices are. If users press this key twice, they 
receive a full screen of help information that 
describes in greater detail the current function 
and the various ways one can proceed. This help 
is context sensitive in that several different help 
messages and screens are available from each 
work screen; the message or help screen displayed 
is dependent upon where the user's cursor resides 
at that point in time on the screen (see Figures 3 
and 4 for examples). 

Users are made aware of the existence of the 
online help from screen prompts and the printed docu- 
mentation. In addition, online help is discussed and 
demonstrated in the hands-on workshops provided for 
the system. 

THE USERS 

Approximately 350 Laboratory employees use the 
Salary Review System. The majority are managers 
(275), with the remainder being their administrative 
assistants and the Laboratory's personnel department 
staff. Because Los Alamos National Laboratory has an 



extensive scientific computing network, many of the 
Laboratory managers have extensive computer literacy 
on command language systems. In addition, many 
have recent experience using personal computers as 
administrative tools. A significant percentage of the 
population (20%). however, have had either little prior 
exposure to computers or only experience with elec- 
tronic mail and word processing systems. 



METHODOLOGY 

User behavior has been monitored three ways: 
direct observation during workshops and during actual 
use, logging of consulting phone calls, and interviews. 
Information was sought to answer the following ques- 
tions: 

• What type of training .tiost effectively produces 
the greatest amount of learning for these users? 
What are the preferred learning styles of these 
users? 

• How frequently do users access online help or per- 
ceive they access online help? 

• How many consulting calls received could ^ave 
been answered by online help, and what learning 
styles do the frequent callers display? 

• How can the online help be modi&ed to better 
meet user needs? 

• Was the time invested in generating the extensive 
online help cost effective? 

Direct Observation During Workshops: Some students 
from each workshop (total 50) were observed to see if 
they spontaneously used the online help when they 
were in trouble, if they had even been aware of the 
availability of online help (learned through the verbal, 
screen, and printed prompts), and if the online help 
accessed answered the users' questions. 
Direct Observation During Actual Use: On occasion, our 
user services staff has the opportunity to observe users 
using the system to perform the task of salary review 
or report generation. Observations here were the same 
as in workshops, but the setting was more natural for 
the users and thus a better indication of whether the 
online help would really be useful. 
Consultants 1 Log: Users may call our consulting office 
during working hours to obtain answers to questions 
r-n data or usage and to report problems. The percen- 
tage of problem versus usage questions is noted. For 
some usage calls, it is noted whether the question could 
have been answered with the online help. 
Interview Because user satisfaction is a high priority 
for our information system, the user services staff reg- 
ularly discussed system design and efficiency with 
users. During many of these discussions, both in per- 
son and by phone, users were asked if they use the 
online help and if so how often and whether it was 
helpful. 
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MAIN MENU 



YOU HAVE THE FOLLOWING OPTIONS: 

SIA : Distribute SIrt 

ROS : Roster worksheet 

CWA * Change write access 

CMS : Change management structure 

RRP : Enter raise report menu 

UID : User identification 

EAC t Enter the authority control nenu 

BUL : Salary review system bulletin beard 

MSG s Messages 



HE : HELP 

LO r LOG OFF SYSTEM 
ENTER THE DESIRED OPTION: HE 

SI* 



Figure 1: Access lo Fixed-Format Help. 



SIA : Distribute SIA 

This screen allows the signon manager to: 

- Allocate SIA to subordinate manager* 

- View subordinate managers and their employees 

ALLOCATING SIA - the signon manager may distribute SIA to his/her 

subordinate Managers. In doing so the ALLOCATED and RESERVE 
amounts are calculated every time a TOTAL allocation for a manager 
is entered. 

VIEWING SUBORDINATE MANAGERS AND THEIR EMPLOYEES - the signon manager may 
view any one of his/her subordinate manager's managers and also a 
subordinate manager's employee's via the ROSter Worksheet screen 



The Salary Review System Quick Buide explains each function in more detail. 



KEY CONTROLS: 

PRESS THE RETURN KEY TO CONTINUE 



Figure 2; Example of a Fixed-Format Help Screen. 



Fl 4-081 D LOS ALAMOS NATIONAL LABORATORY SALARY REVIEW SYSTEM 

RAISE REVIEW REPORT MENU (RRP) 



10/10/B5 



PSR i Produce Standard Report 
PGR : Produca Customized Report 



ENTER THE DESIRED PROCESS! 
(enter PSR or PCR) 



NEXT TASK: (valid tasks arei SI A, R0S 9 CWA, CMS, RRP, EAC, BUL, MSG, HE, LO, MA, EX ) 

VALID KEYS ARF* l <PF2>| » <CTRL-E>, < RETURN > 

[Enter one of "the eel act i on types^^i spi ayed o n the~sc r6an J 



Figure 3: Example of One Line of Context-Sensitive 
Help: One Press of Help Key. 



PSR HELP SCREEN 



TO SELECT ONE OP THE STANDARD REPORTS - this option providee a Structured 
Series report, a Staff Member report, a Jobcode report, and 
an Alphabetical report. 

- TAB to the desired report and press the RETURN key 

- the SOR option may be selected if you wish a different 
order sequence other than the default 

- select the CFL option and press the RETURN key 
(this will create the report) 

- enter NRP to select a report name 

- to find the status of the report repeat the first step 
above 



- if the report is available enter VRP in the ENTER SELECTION 
field ( thie will display the report at the terminal) 

- press the CTRL key and the E key for the RRP RAISE REVIEW 
REPORT MENU 

KEY CONTROLS I 

PRESS HELP KEY (PF2) FOR VIEWING REPORTS INFORMATION! RETURN KEY FOR PSR SCREEf 



Figure 4: Example of Full Screen of Context-Sensitive 
Help: Two Presses of Help Key. 



Measures were all qualitative. An additional 
quantitative method of monitoring user behavior 
would have been useful: a software monitor which 
displays the frequency and context of use of the online 
help. Unfortunately, this was not done because of the 
lack of time to develop such a monitor and the over- 
head of such monitors resulting in a slowing of 
response time. The priorities of this system were meet- 
ing the schedule and providing a new system for 
managers. Although the lack of quantitative results 
does detract from the predictability of the results of 
this study, this study does represent a real-world 
situation where ongoing qualitative measures can 
significantly impact future work. The evaluation 
methodologies used here are commonly used in the 
fields of human factors in computing and cognitive 
psychology. 



RESULTS 

Learning Styles 

Observations of users' preferred methods of 
learning how to use the Salary Review System revealed 
three preferred styles: 

1. DISCOVERY LEARNING STYLE: Some users 
preferred to learn how to use the system com- 
pletely on their own, with no formal training. 
These users log onto the system and browse, 
showing little computer anxiety. They tried the 
different menu options to see what happened and 
used printed documentation as a reference when 
they could not obtain answers from the screen. 
Use of the phone consulting service was reserved 
for questions not answered in documentation. 
These users tended to be somewhat impatient in 
their learning, desiring to learn system features 
only as they needed to use them. 

2. GUIDED DISCOVERY LEARNING STYLE: 
These users preferred to learn by starting to use 
the system in a structured computer classroom or 
one-on-one training situation, but after a very 
short time learning the basics (15-30 minutes) 
they became impatient and wanted to discover 
more about the system on their own. They 
referred to the printed documentation step-by- 
step and then subsequently used it as a reference. 

3. STRUCTURED LEARNING STYLE: These 
users would not begin using the system without 
formal classroom training. During the training 
they followed instructor directions carefully and 
referred to printed documentation when they 
were asked to do so. They discovered little about 
the system on their own, so thorough training on 
all system features was necessary. They often 



had difficulty finding the information they needed 
in the documentation and relied heavily on phone 
consulting after formal training. 

User preferred learning style was determined through 
interviews and direct observation, therefore the 
categorization is subjectively based. 

Frequency of Access of Online Help 

Observations revealed a relationship between 
learning style and frequency of access of online help 
(see Table l). Users with a guided discovery learning 
style seemed more willing to access online help than 
those with a discovery learning style. In turn, the 
structured learning style users seemed to access the 
online help the least. 

The guided discovery learning style users learned 
of the availability of the online help through trainer 
announcements and documentation, and subsequently 
used it in addition to printed documentation. Inter- 
views revealed that the discovery learning style users 
frequency of access of online help seemed dependent 
upon their prior preference of printed versus online 
documentation. A study performed at Los Alamos on 
a different help system suggested that users who were 
previously not familiar with online help tended to rely 
heavily on printed documentation, if they could 
readily locate it. whereas those who had previously 
used online help systems successfully tended to use 
online help more than printed documents (Stoddard et 
al., 1985). This same pattern could also be operating 
here, but further study is necessary to verify this 
hypothesis. 

The structured learning style users revealed some com- 
puter anxiety, so they seemed reluctant to press any 
function keys for fear of making an unrecoverable 
error. This tendency seemed to disappear as the user 
became more comfortable with the system and if. on a 
one-on-one training basis, online help was frequently 
accessed and proven useful. 

Context-Sensitive Versus Fixed-Format Help Fre- 
quency of Use 

Of the two types of help available, the context- 
sensitive help seemed to be most accessed and most 
useful to users. The fixed-format help provided very 
general information on functionality of basic system 
options from the main menu. The method of accessing 
it is somewhat awkward in that one cannot browse 
sequentially through the help screens for all the 
options, but rather must separately request help (HE) 
and then the option name. The same information is 
easily browsed in the printed documentation. 
Discovery learning style users tended to find out about 
the function of opt^ns by entering the option itself to 
see what happened, whereas the guided discovery and 
structured users relied on description from the training 
or the printed document. 
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TABLE 1: Relationship of Learning Style to Help Accessed 



Learning 
Stvle 


Job Category 


Frequency of 
HELP Access 


Phone 
Consulting 


Preferred 
TraininR 


Printed 
Document 


Discovery 


more techni- 
cal managers 


moderately 
high 


low. only 
for non- 
documented 
problems 


self-study 


used as a refer- 
ence 


Guided Discovery 


more non- 
technical and 
technically 
oriented ad- 
ministrative 
assistants 


high 


moderate 


formal, fol- 
lowed by 
self-study 


for initial 
learning and as 
reference 


Structured 


more admin- 
istrative as- 
sistants than 
managers 


low 


high 


formal, fol- 
lowed by 
phone consult- 
ing 


as a reference 
only if guide 
found usef ul 
durin&class 



Context-sensitive help tended to be accessed when 
users were in trouble and did not know what to do 
next. Once it was accessed and was found to be useful 
for troubleshooting, users tended to remember to use it 
subsequently. Users who became accustomed to using 
the system without using the help key. however, 
tended to continue this pattern and not use it. The 
feature of obtaining an entire screen full of help upon 
pressing the help key twice was less frequently used. 
User interviews revealed that, in spite of its being 
documented, it seemed to be difficult to remember and 
illogical to press the help key twice when in trouble. 
Users had not seen this feature in other systems they 
had used, but expressed appreciation for being rem- 
inded about it in the interview. Thus, in spite of the 
fact that the users interviewed had been told of this 
feature during formal training, they rarely remem- 
bered it during independent use. 



SUMMARY AND IMPLICATIONS 

For the purposes of answering our initial ques- 
tions, the results of this study suggest the following; 

• Fixed-format help accessed from the main menu Is 
less effective for user learning than context- 
sensitive help. It could be made more useful by 
making it easier to browse; that is, not require the 
user to type in each menu option when asking for 
HELP and offer all the help screens in succession 
(with the ability to return to the main menu at 
any time). 

• The feature of the double press of the help key for 
obtaining more in-depth help was unnutural, 
although the full screen of help was judged by 
users as useful once they remembered how to get 
it. The interface should be modified, therefore, so 



that the full screen of help will be provided 
instead of one line of help when the task at hand 
is sufficiently complex. 

• Printed documentation should be modified to 
include actual examples of help screens and mes- 
sages. 

• Formal training should Include more frequent 
demonstrations of the online help during the 
hands-on session, 

• When training is advertised for the system, the 
extensive online help feature of this system (com- 
bined with printed documentation) should be 
announced as well as the hands-on workshops to 
respond to the many users with discovery learn- 
ing styles. 

Several hypotheses can be formulated here that 
relate to the use of online help as a form of embedded 
training: 

1. The frequency of access of online help Is 
directly proportional to the degree to which the 
online help is proven useful in demonstrations, 
training, printed documentation, and prior use of 
online help. 

2. The degree of usefulness of online help for 
learning a new system is directly related to the 
degree of naturalness of access to the help. 

3. Learning style is an important variable when 
designing embedded training. Learning style is 
dependent not only on prior computer experience 
but also on a users job and prior experience with 
online and printed computer documentation. 
Since it is experience-based, however, this variable 
must be viewed as dynamic within an individual. 



8 



ERIC 



4. if a system is well documented and has an 
easy-to-use interface, and if the user population is 
dominated by users with discovery or guided 
discovery learning styles, context-sensitive help 
may replace the need for formal training or more 
complex, structured computer-based training 
packages. 

These hypotheses are useful on several counts. 
We can use them in our context for future planning of 
online help and for screening users for formal training. 
They are useful in other contexts because they suggest 
that the specific design of embedded training depends 
upon the dominant learning style of the user popula- 
tion. Thus, before embarking on a project to develop 
embedded training, project developers need to include 
criteria not normally examined in an audience analysis. 
These hypotheses are also usef ul to the research com- 
munity in that they provide several specific areas 
where experimental studies could have a significant 
impact on the direction of embedded training. Our case 
study also demonstrates that commonly used evalua- 
tion methodologies are effective in collecting user 
behavior information for systems already in produc- 
tion. 

The degree of needed sophistication of embedded 
training is a critical decision for the cost-benefit 
analysis in system ■ development. As both Walker 
(1985) and Kearsley and Seidel (1985) point out. 
highly user responsive context-sensitive help requires a 
large effort, and instructional designer involvement at 
an early stage is required. Our case study demon- 
strated, however, that help which is context sensitive 
(but not "intelligent") does not require major increases 
in software development resources and yet it does seem 
to have a positive Impact on ease-of-learning and ease- 
of-use. The recognition that the development of online 
help can be cost effective if integrated into other phases 
of the software life cycle is not new (e.g., Relies et al.. 
1981), and yet demonstration that embedded training 
will further enhance cost effectiveness is not fully 
recognized. 
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